1. Field of the Invention
This invention relates to a user interface for a computing device; in particular a touch-based device such as a smartphone or tablet.
2. Description of the Prior Art
Today, we use many different types of computing devices (such as TVs, laptops, tablets, phones). Their interfaces can be very different, even when from the same company. This makes it harder for users to transition rapidly between these types of devices; it requires users to learn new ways of interacting with these devices, which can be frustrating, especially for users who are less technically sophisticated, or are simply less willing to invest time and effort into learning new interaction skills. It also makes the process of designing these types of devices slower and riskier than necessary, since new interaction design approaches have to be conceived for each new type of device. And it makes it more difficult for developers to write applications for each of the devices, because they cannot reuse very much of their software for each different form factor. In part, this inefficiency arises because the underlying operating systems across these various kinds of devices are not common or shared; further, if one takes a user interface designed for a large device, such as a laptop, and presents it on a smartphone, the result is unusable - the challenges facing interaction designers working with the small screen of a typical smartphone are very considerable.
Overall, the interaction design problems creating a single unified family of interfaces that (a) enables fast one-handed operation of a smartphone and (b) will scale effectively from smartphone to tablet, where two-handed touch interaction is typical, (c) scales smoothly to accommodate keyboard and pointer input typical of a PC, and (d) scales to a TV form that is usable with a simple remote control, are very considerable and have not been successfully solved until now.
This patent specification describes not only various ideas and functions, but also their creative expression. A portion of the disclosure of this patent document therefore contains material to which a claim for copyright is made and notice is hereby given: Copyright Canonical Limited 2012 and 2013 (pursuant to 17 U.S.C. 401). A claim to copyright protection is made to all screen shots, icons, look and feel and all other protectable expression associated with the user interface and user interaction illustrated and described in this patent specification.
The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but reserves all other copyright rights whatsoever. No express or implied license under any copyright whatsoever is therefore granted.
Summary of the invention
In a first aspect, there is provided a smartphone, tablet or other computing device comprising:    (a) a touch sensitive display;    (b) one or more processors;    (c) computer memory;    (d) an operating system computer program stored in the computer memory and configured to be executed by the one or more processors and including instructions for detecting a swipe gesture, and, in response to the detected swipe gesture, displaying a search bar.
This document will describe innovations in the following areas:    A. Edge Gestures    B. Ranged gestures    C. The Welcome Screen    D. The Unlocking Mechanism    E. The Ubuntu search experience    F. The Home Screen in Ubuntu    G. The Ubuntu ‘Side-Stage’    H. Scalable, Responsive Ubuntu UI
In this section we summarise those innovations.
A. Edge Gestures
A central invention in the Ubuntu user experience (UX) is design of the role that each edge plays in the interaction of system, settings, applications and controls. In the Ubuntu UX, the layout of the screen is designed to ensure:                smooth scaling of the experience from phone, to tablet, to PC and to TV        productive touch interaction that maximises the space on the screen available for content and applications, and eliminates the need for physical or software ‘system buttons’        
The best expression of that design is ‘edge magic’. In the Ubuntu UX, and especially in the touch versions of the Ubuntu UX, the edges of the screen play a crucial, distinctive and innovative role.
A.1 Application Launching or Switching is Driven from Left and Right Edges
A swipe in from the left edge is always used to access applications. This is easy in both phone (one-handed) and tablet (two-handed) interfaces, it offers more space for applications than the bottom edge of a phone (like WebOS) and it consumes relatively little of the screen compared to a horizontal dock at the bottom of the screen (like MacOS or Windows 7). Favourite and running applications are directly accessible through the ‘Launcher’ which is a vertical bar of application icons accessed through the left edge (e.g. a short swipe in from the left edge). In this document, we will refer to the Launcher as the ‘Launcher’, ‘launcher’ and ‘app launcher’. More applications are reachable through the ‘Home Screen’ apps page (which is a grid of apps on the device, indicated which apps are running, which are closed; and also showing apps of potential relevance to the user can which can be obtained from a server). This Home Screen apps page can also be accessed directly from the left edge (e.g. by a longer swipe in from the left edge). So the left edge serves as a consistent gateway to any application. The launcher bar includes also an icon for the ‘Home Screen’; the Home Screen shows icons for frequently used applications, calls the user has recently made or received (or missed, so the user can call them back easily by simply touching the item listing the missed call to directly initiate a call back to them). The Home Screen is fully customizable too, so it can include for example ‘Recently Added Music’; New Film Releases; On-line Radio; Wikipedia; on-line stores etc.
A swipe in from the right edge, by contrast, accesses previously-used but still running applications. It provides a convenient way to return to the previous application. The use of opposite edges for ‘switch to the next app’ and ‘switch back to the previous app’ (namely, a swipe in from the left edge to access the Launcher bar, then the user's finger moving up or down to release on a specific app icon in the Launcher to switch to that app; conversely, a swipe in from the right edge to switch back to the previous app) is distinctive to Ubuntu and provides a usability benefit: people quickly learn the behaviour, it feels natural.
A.2 System Settings and Status are Reflected on the Top Edge
The top edge hosts a set of status indicators, and provides a means to access system-wide settings and features (typically icons for network, battery, clock and calendar, sound etc. are placed on the right side of the top edge) as well as system or system-wide capabilities (such as messaging and search; icons for these are typically placed on the left side of the top edge).
A.3 Application Controls are Accessed Through the Bottom Edge
Ubuntu uniquely defines the bottom edge of the screen as the way to access controls for the application—typically, these controls are implemented in a variety of forms, specific to each application: for example, playback controls and a timeline in a video player application; a toolbar with action buttons on an image editor application; a game menu for a gaming application, are all examples of controls that can be included in a bar, window or region that is revealed by a short swipe up from the bottom edge.
The consistent use of an edge as a gateway to application controls enables application developers to remove those controls from their interfaces by default, because users will know there is a standard way to access such controls. That has the benefit of giving more space for application content, making Ubuntu feel more spacious than previous systems on small screens like the phone.
The controls in question can be buttons, or they can be voice controls. For example, the Ubuntu HUD, which is a searchable command interface, is also invoked through the bottom edge. The HUD allows the user to access controls for an application which cannot fit into a simple touch toolbar or other chrome-based (‘buttons on the screen’) interface. The mechanism is called the HUD (‘head-up display’), because it displays information to the user without forcing them to shift from their main focus of interest, analogous to a conventional HUD.
This document is written from the perspective of left-to-right language users. In the case or right-to-left languages, the UX might be horizontally reversed (so left becomes right).
The design of edge behaviours and conventions is one of the key innovations at the heart of Ubuntu UX innovation for touch devices. It sets Ubuntu apart from systems that have come before, even where those used the edges for other purposes. For example, the Nokia N9 used a generic edge gesture to return to the home screen, and Palm WebOS phones used the bottom edge to access favourite applications, but Ubuntu provides substantially richer patterns of use for the scarce resource of screen edges and combines them in unique ways to create a system that is more usable, with richer functionality, than any previous system.
B. Ranged Gestures
Another significant innovation in the Ubuntu UX for touch devices is the use of ranged gestures, by which we mean a gesture which can have a range of diverse outcomes depending on where it ends or how far it progresses. We don't just mean a progressive gesture, like sliding a control along a slider, where the result can be a variable. We mean a gesture which could have any number of quite different results depending on the extent of the gesture (such as, distance, length, direction, path, start-pint, end-point).
Ranged gestures are important in the Ubuntu UX because they allow a single resource, the edge of a screen, to be a gateway to a range of different outcomes or capabilities. They also allow the user to navigate through all of those capabilities in a faster and more fluid fashion, where multiple ‘steps’ or ‘taps’ or ‘swipes’ can be combined into a single, directed gesture by more sophisticated users.
Ranged gestures can contain stages (extending the gesture takes you through a series of modes, ending the gesture selects a particular action from the mode you are in at that point, or enables you to continue selecting with a subsequent gesture). They can also contain bifurcated pathways, where the gesture can flow in a series of different directions to expose different options.
B.1 Application Edge (Left Edge) Ranged Gestures
A swipe through the left edge is a ranged gesture with several stages.
The initial left edge swipe will simply reveal the launcher. The initial hint of the launcher can be undone by reversing the swipe back over the edge, or releasing very early in the hint. A small swipe, however, is enough to show the launcher with favourite apps. Releasing at this stage leaves the launcher visible, to be manipulated with further touches or swipes or taps.
A longer swipe will select the nearest application on the launcher (i.e. the application for the icon closest to the touch path of the user's finger, plus the Home Screen icon), and moving the contact or touch point up and down allows the user to select a particular application other than the one closest to their finger when they crossed the edge initially; an app which has been chosen (but not yet actually selected) will typically be shown with a larger icon that its neighbouring icons. This approach allows for fluid selection of an app to launch or switch to on the Launcher. Releasing the touch over a chosen icon for a specific app will select that application.
Pausing over an icon in the Launcher in the ranged gesture will show the Quicklist of that application, allowing the ranged gesture to bifurcate and pursue the items in that Quicklist. This allows for fluid access to a particular item in a particular application Quicklist in a single gesture starting with the edge swipe. A Quicklist for an application is a specific feature in an app that enables a user to rapidly access a feature or function or control for that app. For example, if the app is an e-mail messaging app, then the Quicklist could be to start composing a new e-mail message.
If, rather than pausing on an icon, the gesture ranges past the launcher to the right quickly, then a new stage is reached, which reveals the Apps page of the home screen. This stage enables access to applications which are not in the launcher (favourites or running) or not even installed.
So, ranged gestures allow the left edge to provide a comprehensive gateway to applications on the device—favourites, running, installed and not-yet-installed.
B.2 Right Edge Ranged Gesture
The right edge has a ranged gesture which controls access to the previous focused application (namely, the application the user was previously using) on touch devices such as the phone, or the tablet with visible ‘side-stage’ (the ‘side-stage’ is described more fully later in this document).
The initial stage of the gesture, swiping through the right edge, will show a hint of the previous application, or a hint that there was no previous application focused. Releasing in this early portion of the gesture would have no effect, returning the system to the state it was in before the hint—serving to prevent an unintended action—switching to a previous application.
Continuing the gesture would commit to showing that previous application as the focused app on the phone, or the focused app in the tablet ‘side-stage’ (see later for a description of how the ‘side-stage’ can display apps independently of what is shown on the rest of the display).
On a tablet where there is a side stage, the right edge swipe initially hints at the side stage, then commits to showing the side stage as an overlay, and ultimately commits to splitting the screen between main and side stages.
B.3 System (Top) Edge Ranged Gesture
The top edge provides access to system services, settings and searches, through a ranged gesture.
What is unusual about the Ubuntu UX top edge ranged gesture is that it allows very rapid and fluid access to the information that backs up the core system indicators at the top edge, right side of the screen.
On existing phones, it is difficult to interact with the system indicators. On iPhone iOS6 and earlier, you need to find and then switch to a special application, the settings app, which provides access to system settings (and, as it happens, application settings).
On Ubuntu, it is very easy to reach settings through the top edge. They are on the right of the top edge, so an initial swipe through the right part of the top edge reveals system indicators if they were off-screen (for full-screen applications) and selects the closest. That can be reversed by reversing direction and swiping back up off the screen through the top edge. At this point, the ranged gesture can be bifurcated—moving left or right (without lifting the finger) enables the selection of a particular system indicator, and moving downwards confirms the focus on the selected indicator.
This enables:                Fast access to any particular system setting or indicator,        Rapid inspection of a variety of system information (for example—‘Which network am I on, and what new messages have arrived?’)        
In addition, the left part of the top edge is dedicated to search. So initiating the gesture on the top left of the screen provides access to a range of system search options.
B.4 Application Controls (Bottom) Edge Ranged Gesture
The bottom edge in Ubuntu provides access to and control over the application controls.
Given that applications might follow very different designs for very different purposes, the nature of those controls can vary. But in general, a bottom edge swipe should hide and reveal controls of the appropriate form for the application. Those controls might be a toolbar, or a playback control (play, pause, back, forward), or a timeline, or might involve change the status of the application (for example, pausing a game and presenting menu options).
The bottom edge also has a ranged gesture—the initial swipe through the edge hints at controls which can be reversed by changing direction and going back off the bottom edge, or releasing before the hint turns into a committed reveal of the controls. Once controls are revealed, the gesture can be continued, and new stage is revealed, which includes the ability to launch the HUD (a text and voice-searchable control system for the application). Releasing the gesture over the HUD icon, or one of the other ranged end points of the gesture, commits to that outcome.
C. The Welcome Screen
When the user turns on the device, the device provides a beautiful and refined reflection of the user and their activities in a live welcome screen, that is a data visualisation of data that is personal to the user or specific to the device.
C.1 Data Visualisation Rather than App Info or Wallpaper
It is innovative to display a data visualisation on the welcome screen. In existing phone and tablet devices, it is common to have a personal image (‘wallpaper’) on the lock screen, or in some cases to represent the status of an application (‘5 new messages’). What is also innovative about Ubuntu is that the data visualisation can be a synthesis of data from multiple sources—from applications, online services, and the device—rendered in an artistic fashion.
The data visualisation may or may not be interactive, and may or may not trigger the launch of an application or content based on that interaction, thereby unlocking the device.
The data visualisation may be combined with or be part of a personal artwork, or wallpaper, and may draw its palette from the artwork so as to combine visualisation and wallpaper in an aesthetically pleasing way.
The data visualisation may be selectable from a range of options, or chosen randomly, or updated dynamically by the operator, OEM (or any other partner) or Ubuntu itself. It may also be possible for users to design and install their own visualisations.
D. The Unlocking Mechanism
This ‘welcome screen’ serves a similar purpose to the ‘lock screens’ found on existing phones and tablets—it prevents inadvertent disclosure of sensitive information, and it prevents inadvertent input when a device is unattended and perhaps being touched inside a handbag or pocket. However, it is distinctive and innovative in a number of ways:                In existing lock screens, the device behaves very differently when locked than when unlocked. For example, buttons that normally take you to the home screen cannot do so when the device is locked. On the Ubuntu phone and tablet, which use the edges of the screen to navigate between applications, controls and settings, the welcome screen behaves just like any other application. To ‘unlock’ the device you would use (namely, swipe in from) one of the edges to navigate directly to the application or setting that you want. On a traditional phone, you need to unlock before being able to do anything. On Ubuntu, you can start doing anything, and it will only ask for access control when needed.        It is possible to launch any favourite application when the welcome screen is showing. In existing lock screens, there would be a few (possibly customisable) applications that can be launched when locked, but the mechanism to launch them would be very different when locked than unlocked. For example, on the iPhone 4 there is an ‘unlock’ mechanism (a sliding switch) and a mechanism to launch the camera when locked, but those are completely different than the normal mechanism to launch the camera application, which takes place from the home screen. The Ubuntu approach is more consistent between locked and unlocked states.        On the Ubuntu phone and tablet, based on edge gestures, there is a clear distinction between unlocking to launch a favourite application, unlocking to the home screen, unlocking to the application which was running when the phone was locked, and unlocking to modify a system setting. On existing lock screens, ‘unlocking’ might take you to the home screen, or to an application, depending on what you were doing when the phone was locked.        